Traffic Signs Recognition and Classification Using Convolution Neural Network (CNN)¶

Deep Learning Project¶

UB ID:¶

50442703

Name:¶

Venkata Rohil Wardhan Kancharla

Jupyter Notebook Drive Link:¶

https://drive.google.com/file/d/1R194oIauIKYKXVMFI21Zcw9pCZag-mtY/view?usp=share_link

Notes for Code Execution -¶

  • First import the kaggle dataset by running the first 5 lines of code and entering the username and key provided.
  • Now place, 'train.pickle', 'data2.pickle', 'labels.csv' files in the same folder as the '.ipynb' file.
  • Run the code starting from inputs using Jupyter Notebook.
  • Do not use colab as RAM usage is high.
  • General Perspective
  • Problem Statement
  • Traffic Signs
  • Traffic Recognition System
  • Description of Dataset
  • State-of-the-art Algorithms
  • Results
  • Future Work
  • References

Every traffic sign is designed in a way where it is visible irrespective of the background or time of the day. Different traffic signs vary based on thier shapes and colours with a shape or number present on the sign. Traffic sign recognition is an important part of driver assitance and to make self-driving vehicles. Classification of traffic signs is an important part of Advanced Driver Assistance Systems (ADAS) which allows the vehicles to Self-drive with particular set of rules and within a spped limit.

Problem Statement¶

Traffic Signs are crucial for following proper rules on roads to ensure safety on the road. Self-Driving cars is an upcoming technology in the car industry which requires accurate functioning of Traffic Sign Recognition (TSR) System. Currently for the Self-Driving technology one of the core technologies is Traffic Sign Recognition(TSR) alongside Advanced Driving Assistant System(ADAS). TSR is also used in all cars launched after 2015 to alert the driver of the speed limit on the road. Road signs are recognised by the camera mounted in front of the car. In Germany, a TSR benchmarking competition is held to support the industrial development.

Traffic Signs -¶

Traffic Signs are road signs placed on the sides of roads to instruct or inform the road users of the road conditions or the rules implemented on the road.

There are different types of traffic signs like:

    Warning Signs
    Prohibition Signs
    Mandatory Signs

Warning Signs: To warn the road users of any hazardous conditions on the road

image-2.png

Prohibition Signs: Actions that are not permitted for that road

image-2.png

Mandatory Signs:

image-2.png

Traffic Sign Recognition Consists of:¶

1) Detection: Identifying a traffic sign and ignoring the background at higher speeds

2) Classification: Differentiating the different traffic signs based on the colour and shape of the sign and then identifying the sign based on what is displayed

image.png

Description of the Dataset¶

  • The dataset taken is a preprocessed data of the traffic signs that have been shuffled and the data has been normalised.
  • All datasets taken are in the form of pickle files.
  • Train, Test and Valid are the initial pickle file datasets for the training, testing and validation of data when machine learning models are built for classification.
  • Pickle files show the data as values instead of images for better understanding of data.
  • The images of the dataset are in 2D form.

Different types of traffic signs are defined in the "label_names.csv" file

image.png

Continued -

image.png

Different types of preprocessing are performed on the data and saved as various pickle files.

image.png

The initial train dataset before preprocessing is done. This data is a dictionary with four keys.

image.png image-2.png

The four keys of the train data are defined as -

'features' - Is a 4D array with raw pixel data of the traffic sign images,
                     (number of examples, width, height, channels).
'labels'   - Is a 1D array containing the label id of the traffic sign image,
                     file label_names.csv contains id -> name mappings.
'sizes'    - Is a 2D array containing arrays (width, height),
                     representing the original width and height of the image.
'coords'   - Is a 2D array containing arrays (x1, y1, x2, y2),
                     representing coordinates of a bounding frame around the image.

Method 'astype' is used to convert ndarray from int to float.

The size of the train data for each keys is displayed -

x = d['features'].astype(np.float32)   # 4D numpy.ndarray type, for train = (34799, 32, 32, 3)
y = d['labels']                        # 1D numpy.ndarray type, for train = (34799,)
s = d['sizes']                         # 2D numpy.ndarray type, for train = (34799, 2)
c = d['coords']                        # 2D numpy.ndarray type, for train = (34799, 4)

The dataset used for the recognition and classification of data for this project is data2.pickle.

image-3.png image-4.png image-5.png

The data2.pickle file contains all the all the train, test and validation data used for the deep learning model.

Shape of train, test and validation data -

image.png

Some examples of training data in grid form -

image.png

State-of-the-art Algorithms¶

  • The main deep learning model used for this project is Convolution Neural Network (CNN).
  • The activation function used is 'ReLu' and 'Softmax'
  • The optimizer is 'Adam'
  • A learning rate of "lambda x:1e-3*0.95**(x+epochs)" is taken

Convolution Neural Network¶

  • A Convolutional Neural Network (ConvNet/CNN) is a Deep Learning algorithm that can take in an input image, assign importance to various aspects/objects in the image, and be able to differentiate one from the other.
  • The preprocessing of CNN is much lower compared to other neural networks.

Flow Chart of CNN -

image.png

Activation Function¶

  • Activation Function is used to get the output of a node.
  • It is used to determine the output of a neural network which is shown as 0 or 1.
  • The role of the Activation Function is to derive output from a set of input values fed to a node (or a layer).
  • Without defining activation function, every model works as linear regression model.

Different Types of activation functions mainly used are -

  • Binary Step
  • Linear
  • Sigmoid
  • Tanh
  • ReLU
  • Leaky ReLU
  • Parameterised ReLU
  • Exponential Linear Unit
  • Swish
  • Softmax

Optimizer¶

  • An optimizer is a function or an algorithm that modifies the attributes of the neural network, such as weights and learning rate.
  • optimizers are used to adjust the parameters for a model.
  • The purpose of an optimizer is to adjust model weights to maximize a loss function which is used as a way to measure how well the model is performing.
  • Thus it helps in reducing the overall loss and improve the accuracy.
  • Most common optimizer used is "Adam".

Different types of Optimizers -

  • Momentum
  • Nesterov
  • Adagrad
  • Adadelta
  • RMSProp
  • Adam
  • Nadam

Learning Rate¶

  • Learning rate is a hyperparameter used in the training of neural network that is positive usually in the range of 0-1.
  • The learning rate controls how quickly the model is adapted to the problem.
  • Learning rate is also used to reduce the loss of a neural network model.

Effect of Various Learning Rate -

image.png

Model -¶

Summary of the Model -

image.png

Initial result -¶

During the training of the data only using CNN network model, validation accuracy is close to zero.

This occurs due to overfitting of data.

image-3.png

Plot of the Initial Trained Model -

image-3.png

Filters¶

  • A filter acts as a single pattern, which, when convolved across the input, finds similarities between the stored template & different locations/regions in the input image.
  • Multiple filters are used when different features are to be extracted from a single image.
  • Multiple filters helps to apply convolution in a batch.

Model 1 With Filters -¶

  • Different filters used for this project are 3, 5, 9, 13, 15.
  • These filters are taken as "{0:d}x{0:d}" form based on the number of filters.
  • Filters are applied to the CNN model remove the overfitting of data and improve the feature extraction in order to improve accuracy.

image.png

Results -¶

Accuracy of the data after filter is applied -

image-3.png

Training Accuracy of the Data is how accurately the data has been trained.

image-3.png

Validation accuracy of the data is the output accuracy. Output accuracy refers to how weel the model has recognised and classified the traffic sign images.

image-3.png

Values of all the Training and Validation Accuracy from the above Results -

image-3.png

Testing Accuracy is used to check how well the model has performed when compared to the validation accuracy.

image.png

Classification Time -¶

  • Classification time is the time taken to classify the images after training of the data.

image-2.png

Output of Some Sampled Data with Filter 3x3 -

image-2.png

With Filter 5x5 -

image.png

With Filter 9x9 -

image.png

With filter 13x13 -

image-2.png

With filter 15x15 -

image-2.png

Model 2 With Filters -¶

  • For model 2, filters 3, 5, 9 are taken.
  • Less filters are taken as convolution layers have been increased.

image.png

Model 2 Summary -

image.png

Results -¶

Results of Model 2 -

image.png

Training Accuracy of model 2 -

image.png

Validation Accuracy of Model 2 -

image.png

Result Values of Training and Validation Accuracy of Model 2 -

image.png

Testing Accuracy of Model 2 -

image.png

Classification Time of Model 2 -

image.png

Output of Model 2 Sampled Data with Filter 3x3 -

image.png

Model 2 with Filter 5x5 -

image.png

Model 2 with Filter 9x9 -

image.png

Detection Sample Output -

image.png

Result Discussion -¶

  • The validation accuracy of model 1 is less than validation accuracy of model 2.
  • Classification time of filter 9 in model 2 is a little more than that of model 1.
  • Runtime of Model 2 is much larger than model 1 as extra convolution layers have been added.

Future Work -¶

  • Self-driving cars are the future of transportation with traffic sign recognition (TSR) as of the core technologies.
  • Letting the driver become vary of the traffic signs through voice assistant in the car after TSR to increase road safety.
  • Detecting of traffic signs under heavy weather conditions.

Problems Faced:

  • Dirt is the biggest issue faced as the car will not be able to detect the sign or even cannot classify the sign based on the symbol present on the sign. Unable to read traffic signs causes unsafe driving behaviours.

  • Under heavy weather conditions like weather or rain, detection of traffic signs is a huge issue.

References -¶

  • A. De La Escalera, L.E. Moreno, M.A. Salichs, J.M. Armingol Road traffic sign detection and classification IEEE Trans. Ind. Electron., 44 (6) (1997), pp. 848-859
  • Á. Arcos-García, J.A. Álvarez-García, L.M. Soria-Morillo Deep neural network for traffic sign recognition systems: an analysis of spatial transformers and stochastic optimisation methods Neural Netw., 99 (2018), pp. 158-165
  • Ren S., He K., R. Girshick, Sun J. Faster r-cnn: towards real-time object detection with region proposal networks Proceedings of Advances in Neural Information Processing Systems (2015), pp. 91-99
  • J. Deng, W. Dong, R. Socher, Li L.-J., Li K., Fei-Fei L. Imagenet: a large-scale hierarchical image database Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, CVPR, IEEE (2009), pp. 248-255

Continued -¶

  • S. Houben, J. Stallkamp, J. Salmen, M. Schlipsing, C. Igel Detection of traffic signs in real-world images: the German traffic sign detection benchmark Proceedings of the 2013 International Joint Conference on Neural Networks, IJCNN, IEEE (2013), pp. 1-8
  • M. Mathias, R. Timofte, R. Benenson, L. Van Gool Traffic sign recognitionâ;;how far are we from the solution? Proceedings of the 2013 International Joint Conference on Neural Networks, IJCNN, IEEE (2013), pp. 1-8
  • Zang D., Zhang J., Zhang D., Bao M., Cheng J., Tang K. Traffic sign detection based on cascaded convolutional neural networks Proceedings of 2016 17th IEEE/ACIS International Conference on Software Engineering, Artificial Intelligence, Networking and Parallel/Distributed Computing, SNPD (2016), pp. 201-206
  • F. Jurišić, I. Filković, Z. Kalafatić Multiple-dataset traffic sign classification with onecnn Proceedings of 2015 3rd IAPR Asian Conference on Pattern Recognition, ACPR, IEEE (2015), pp. 614-618